Reinforcement Learning with Hierarchies of Machines
نویسندگان
چکیده
We present a new approach to reinforcement learning in which the policies considered by the learning process are constrained by hierarchies of partially specified machines. This allows for the use of prior knowledge to reduce the search space and provides a framework in which knowledge can be transferred across problems and in which component solutions can be recombined to solve larger and more complicated problems. Our approach can be seen as providing a link between reinforcement learning and “behavior-based” or “teleo-reactive” approaches to control. We present provably convergent algorithms for problem-solving and learning with hierarchical machines and demonstrate their effectiveness on a problem with several thousand states.
منابع مشابه
Efficient Reinforcement Learning with Hierarchies of Machines by Leveraging Internal Transitions
In the context of hierarchical reinforcement learning, the idea of hierarchies of abstract machines (HAMs) is to write a partial policy as a set of hierarchical finite state machines with unspecified choice states, and use reinforcement learning to learn an optimal completion of this partial policy. Given a HAM with deep hierarchical structure, there often exist many internal transitions where ...
متن کاملSpeeding Up HAM Learning with Internal Transitions
In the context of hierarchical reinforcement learning, the idea of hierarchies of abstract machines (HAMs) is to write a partial policy as a set of hierarchical finite state machines with unspecified choice states, and use reinforcement learning to learn an optimal completion of this partial policy. Given a HAM with potentially deep hierarchical structure, there often exist many internal transi...
متن کاملConcurrent Hierarchical Reinforcement Learning for RoboCup Keepaway
RoboCup Keepaway, originated from the RoboCup soccer simulation 2D challenge, has been widely used as a machine learning benchmark. In this paper, we present a concurrent hierarchical reinforcement learning approach to RoboCup Keepaway. Following the idea of hierarchies of abstract machines (HAMs), we write a partial policy as a HAM from the perspective of a single keeper, run multiple instance...
متن کاملSignals Reinforcement Inputs Sensory Actions Skill Skill Skill
While the need for hierarchies within control systems is apparent, it is also clear to many researchers that such hierarchies should be learned. Learning both the structure and the component behaviors is a diicult task. The beneet of learning the hierarchical structures of behaviors is that the decomposition of the control structure into smaller transportable chunks allows previously learned kn...
متن کاملUsing Abstract Models of Behaviours to Automatically Generate Reinforcement Learning Hierarchies
In this paper we present a hybrid system combining techniques from symbolic planning and reinforcement learning. Planning is used to automatically construct task hierarchies for hierarchical reinforcement learning based on abstract models of the behaviours’ purpose, and to perform intelligent termination improvement when an executing behaviour is no longer appropriate. Reinforcement learning is...
متن کاملذخیره در منابع من
با ذخیره ی این منبع در منابع من، دسترسی به آن را برای استفاده های بعدی آسان تر کنید
عنوان ژورنال:
دوره شماره
صفحات -
تاریخ انتشار 1997